{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "s-Nso4g7VXxy"
   },
   "source": [
    "<a href=\"https://colab.research.google.com/github/microsoft/CameraTraps/blob/master/detection/megadetector_colab.ipynb\">\n",
    "  <img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open in Colab\"/>\n",
    "</a>\n",
    "\n",
    "Link in case the above badge doesn't redirect you correctly: [Open in Colab](https://colab.research.google.com/github/microsoft/CameraTraps/blob/master/detection/megadetector_colab.ipynb)\n",
    "\n",
    "This notebook replaces a previous example by [@louis030195](https://github.com/louis030195). Improvements: updated environment setup, MegaDetector model version and support for mounting Google Drive folders so you can process your own images here."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "vUXNQZtwEYiQ"
   },
   "source": [
    "# Running MegaDetector on camera trap images using Google Colab\n",
    "Put together by Alistair Stewart, Alice Springs, May 2020.\n",
    "@alsnothome\n",
    "\n",
    "For reference please read the [MegaDetector guide on GitHub](https://github.com/microsoft/CameraTraps/blob/master/megadetector.md) and check there for updates. Here we have roughly followed the steps for running under Linux.\n",
    "\n",
    "This notebook is designed to load camera trap image files already uploaded onto Google Drive. If you don't have images already loaded onto Google Drive or just want to see a demo of MegaDetector in action, we also provide code to download some sample images.\n",
    "\n",
    "The steps walk through copying of all of the required model and helper files to the Colab runtime and installing all the required packages. You can then connect to your Google Drive folder and process all of the images in a folder using the MegaDetector saved model. The output is saved in a JSON file - a text based database file whose format is described in this [section](https://github.com/microsoft/CameraTraps/tree/master/api/batch_processing#batch-processing-api-output-format) in the batch API user guide. The detections (as bounding boxes) can then be rendered on your images.\n",
    "\n",
    "The Google Colab instance will only stay open for a maximum 10-12 hrs and after that it will close and any unsaved data will be lost. We recommend saving the JSON output and annotated images into your Google Drive folder for persistent storage."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "9aUlxnm7cnWy"
   },
   "source": [
    "## Set up the Colab instance to run on GPU processing\n",
    "\n",
    "\n",
    "Navigate to Edit→Notebook Settings and select \"GPU\" from the Hardware Accelerator drop-down "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "LUyqKSAWRGNw"
   },
   "source": [
    "## Copy the model, install dependencies, set PYTHONPATH\n",
    "\n",
    "Note: from here on you'll start seeing a mix of code. Most are Linux system commands, rather than Python. The system commands are prefixed by a shebang `!`, which tells this notebook to execute them on the command line."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "ddPlAKHFTn3m"
   },
   "source": [
    "### Install TensorFlow v1\n",
    "\n",
    "TensorFlow is already installed in Colab, but our scripts are not yet compatible with the newer version of TensorFlow. \n",
    "\n",
    "Please follow the next three steps in sequence and do not skip any steps :) If you were not able to follow these, you can reset the runtime by going to \"Runtime\" in the top menu and \"Factory reset runtime\".\n",
    "\n",
    "\n",
    "1. Uninstall the existing version of TensorFlow (this doesn't affect your other Colabs, don't worry)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pip uninstall tensorflow"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "2. Install the older TensorFlow version using `pip`, with GPU processing by specifying `-gpu` and version number `1.13.1`. We also install the other required Python packages that are not already in Colab - `humanfriendly` and `jsonpickle`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "EMEkgpy6T0pr"
   },
   "outputs": [],
   "source": [
    "pip install tensorflow-gpu==1.13.1 humanfriendly jsonpickle"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "3. Importantly, you now need to **re-start the runtime** of this Colab for it to start using the older version TensorFlow that we just installed.\n",
    "\n",
    "Click on the \"Runtime\" option on the top menu, then \"Restart runtime\". After that, you can proceed with the rest of this notebook.\n",
    "\n",
    "Let's check that we have the right version of TensorFlow (1.13.1):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "print(tf.__version__)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "hXn_-PZqTWB4"
   },
   "source": [
    "### Download the MegaDetector model file\n",
    "\n",
    "Currently, v4.1 is avaialble by direct download. The link can be found in the GitHub MegaDetector readme: MegaDetector v4.1, 2020.04.27 frozen model (.pb)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "s5uwmpmaTZMX"
   },
   "outputs": [],
   "source": [
    "!wget -O /content/megadetector_v4_1_0.pb https://lilablobssc.blob.core.windows.net/models/camera_traps/megadetector/md_v4.1.0/md_v4.1.0.pb"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "nmJ6lQX8S4im"
   },
   "source": [
    "### Clone the two required Microsoft git repos\n",
    "This will copy the latest version of the Microsoft AI for Earth \"utilities\" and \"Camera Traps\" repositories from GitHub. These make data handling and running the model easy. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "7qhltAaRSe1W"
   },
   "outputs": [],
   "source": [
    "!git clone https://github.com/microsoft/CameraTraps\n",
    "!git clone https://github.com/microsoft/ai4eutils"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "XQTdfBPZiXiV"
   },
   "source": [
    "We'll also copy the Python scripts that run the model and produce visualization of results to the working directory."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "t4Ns5PjeiTro"
   },
   "outputs": [],
   "source": [
    "!cp /content/CameraTraps/detection/run_tf_detector_batch.py .\n",
    "!cp /content/CameraTraps/visualization/visualize_detector_output.py ."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "2pzfM5Y-iby1"
   },
   "source": [
    "### Set `PYTHONPATH` to include `CameraTraps` and `ai4eutils`\n",
    "\n",
    "Add cloned git folders to the `PYTHONPATH` environment variable so that we can import their modules from any working directory.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "d8vanlgAOlEj"
   },
   "outputs": [],
   "source": [
    "import os\n",
    "os.environ['PYTHONPATH'] += \":/content/ai4eutils\"\n",
    "os.environ['PYTHONPATH'] += \":/content/CameraTraps\"\n",
    "\n",
    "!echo \"PYTHONPATH: $PYTHONPATH\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "JyjEgkCsOsak"
   },
   "source": [
    "## Mount Google Drive in Colab\n",
    "You can mount your Google Drive if you have sample images there to try MegaDetector on or want to save the results to your Google Drive.\n",
    "\n",
    "Once you run the cell below, it will show a URL and a text box.\n",
    "\n",
    "Visit that URL to choose the Google account where the images you want to process live. After you authenticate, an authorization code will be shown. Copy the authorization code to the text box here. \n",
    "\n",
    "Your Google Drive folders will then be mounted under `/content/drive` and can be viewed and navigated in the Files pane.\n",
    "\n",
    "The method is described under this Colab code snippet: https://colab.research.google.com/notebooks/io.ipynb#scrollTo=u22w3BFiOveA. Never give out your account username and password. Read this Colab code snippet to understand how this connection is made and authenticated. There are other ways to connect your Google Drive or upload your data if you do not find this method suitable."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "XYsrTTR7eF0r"
   },
   "outputs": [],
   "source": [
    "from google.colab import drive\n",
    "drive.mount('/content/drive')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Optional: Download Sample Images\n",
    "\n",
    "We install Microsoft Azure's [`azcopy` utility](https://docs.microsoft.com/en-us/azure/storage/common/storage-use-azcopy-v10) which we then use to download camera trap images from the [Snapshot Serengeti](http://lila.science/datasets/snapshot-serengeti) dataset hosted on [lila.science](http://lila.science)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "%%bash\n",
    "# download azcopy\n",
    "wget -O azcopy_linux.tar.gz https://aka.ms/downloadazcopy-v10-linux\n",
    "tar -xvzf azcopy_linux.tar.gz --wildcards */azcopy --strip 1\n",
    "rm azcopy_linux.tar.gz\n",
    "chmod u+x azcopy\n",
    "\n",
    "# copy Snapshot Serengeti images to a local directory\n",
    "DATASET_URL=\"https://lilablobssc.blob.core.windows.net/snapshotserengeti-unzipped/\"\n",
    "SAMPLE_DIR=\"S1/D05/D05_R4\"\n",
    "SAS_TOKEN=\"?st=2020-01-01T00%3A00%3A00Z&se=2034-01-01T00%3A00%3A00Z&sp=rl&sv=2019-07-07&sr=c&sig=/DGPd%2B9WGFt6HgkemDFpo2n0M1htEXvTq9WoHlaH7L4%3D\"\n",
    "LOCAL_DIR=\"/content/snapshotserengeti\"\n",
    "\n",
    "./azcopy cp \"${DATASET_URL}${SAMPLE_DIR}${SAS_TOKEN}\" \"${LOCAL_DIR}\" --recursive"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# this should show 48 images downloaded\n",
    "!ls /content/snapshotserengeti/ -R"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Optional: Download Sample Images\n",
    "\n",
    "We install Microsoft Azure's [`azcopy` utility](https://docs.microsoft.com/en-us/azure/storage/common/storage-use-azcopy-v10) which we then use to download camera trap images from the [Snapshot Serengeti](http://lila.science/datasets/snapshot-serengeti) dataset hosted on [lila.science](http://lila.science)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "%%bash\n",
    "# download azcopy\n",
    "wget -O azcopy_linux.tar.gz https://aka.ms/downloadazcopy-v10-linux\n",
    "tar -xvzf azcopy_linux.tar.gz --wildcards */azcopy --strip 1\n",
    "rm azcopy_linux.tar.gz\n",
    "chmod u+x azcopy\n",
    "\n",
    "# copy Snapshot Serengeti images to a local directory\n",
    "DATASET_URL=\"https://lilablobssc.blob.core.windows.net/snapshotserengeti-unzipped/\"\n",
    "SAMPLE_DIR=\"S1/D05/D05_R4\"\n",
    "SAS_TOKEN=\"?st=2020-01-01T00%3A00%3A00Z&se=2034-01-01T00%3A00%3A00Z&sp=rl&sv=2019-07-07&sr=c&sig=/DGPd%2B9WGFt6HgkemDFpo2n0M1htEXvTq9WoHlaH7L4%3D\"\n",
    "LOCAL_DIR=\"/content/snapshotserengeti\"\n",
    "\n",
    "./azcopy cp \"${DATASET_URL}${SAMPLE_DIR}${SAS_TOKEN}\" \"${LOCAL_DIR}\" --recursive"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# this should show 48 images downloaded\n",
    "!ls /content/snapshotserengeti/ -R"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Lkugt7r3uUEr"
   },
   "source": [
    "## MegaDetector batch processing\n",
    "\n",
    "This step executes the Python script `run_tf_detector_batch.py` that we copied from the CameraTraps repo. It has three mandatory arguments and one optional:\n",
    "\n",
    "1.   path to the MegaDetector saved model file.\n",
    "2.   a folder containing images. If your images were already on Google Drive, replace `[Image_Folder]` with your folder name from Google Drive. If you are using the sample images from Snapshot Serengeti, change `images_dir` to `'/content/snapshotserengeti'`.\n",
    "3.   the output JSON file location and name - replace `[Output_Folder]` with your folder name and `[output_file_name.json]` with your file name.\n",
    "4.   option `--recursive` goes through all subfolders to find and process all images within.\n",
    "\n",
    "You will need to change the image folder path and output file path, depending on your situation.\n",
    "\n",
    "In our experience the Colab system will take ~30 seconds to intialize and load the saved MegaDetector model. It will then iterate through all of the images in the folder specified. Processing initially takes a few seconds per image and usually settles to ~1 sec per image. That is ~60 images per minute or ~3600 images per hour. Limit the number of images in your folder so that all of the processing can be completed before the Colab session ends.\n",
    "\n",
    "If you see the error \"AssertionError: output_file specified needs to end with .json\" then you haven't update the output folder and file name in the line of code below properly."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "pSIH-k0nfi73"
   },
   "outputs": [],
   "source": [
    "images_dir = '/content/drive/My Drive/[Image_Folder]'\n",
    "\n",
    "# choose a location for the output JSON file\n",
    "output_file_path = '/content/drive/My Drive/[Output_Folder]/[output_file_name.json]'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Bsvuux-yhpLw"
   },
   "source": [
    "Here we pass the Python variable value `output_file_path` you specified above to the bash commands below using `$` (double quoting as there are spaces in this path), to run the script. This is so that we can refer to the output file path later for visualization."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "3AOKfviGuTNg"
   },
   "outputs": [],
   "source": [
    "!python run_tf_detector_batch.py megadetector_v4_1_0.pb \"$images_dir\" \"$output_file_path\" --recursive"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "-tHu5WUGDpcd"
   },
   "source": [
    "## Visualize batch processing script outputs\n",
    "\n",
    "Here we use the `visualize_detector_output.py` in the `visualization` folder of the Camera Traps repo to see the output of the MegaDetector visualized on our images. It will save images annotated with the results (original images will *not* be modified) to the `[Visualization_Folder]` you specify here.\n",
    "\n",
    "The scripts take in a number of optional parameters to control output image size and how many are sampled (if you've processed a lot of images but only want to visualize the results on a few) - take a look at the `main()` function in the script to see what other parameters are available."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "iv6ph0l1obhr"
   },
   "outputs": [],
   "source": [
    "visualization_dir = '/content/[Visualization_Folder]'  # pick a location for annotated images"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "en3TbCftkWDE"
   },
   "outputs": [],
   "source": [
    "!python visualize_detector_output.py \"$output_file_path\" \"$visualization_dir\" --confidence 0.8 --images_dir \"$images_dir\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "O0AYUcBlm9BN"
   },
   "outputs": [],
   "source": [
    "import os\n",
    "from PIL import Image"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "AglNEK0goyjA"
   },
   "outputs": [],
   "source": [
    "for viz_file_name in os.listdir(visualization_dir):\n",
    "  print(viz_file_name)\n",
    "  im = Image.open(os.path.join(visualization_dir, viz_file_name))\n",
    "  display(im)  # display() is an iPython method that comes with the notebook"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Next steps\n",
    "\n",
    "Now that you have tried applying the MegaDetector on your own images and assessed its effectiveness, here are some pointers to help you take advantage of the MegaDetector to label your survey images more quickly.\n",
    "\n",
    "\n",
    "### Ways to use the output JSON in my workflow \n",
    "\n",
    "#### 1. Timelapse\n",
    "\n",
    "[Timelapse](http://saul.cpsc.ucalgary.ca/timelapse/pmwiki.php?n=Main.HomePage) is an open-source tool for annotating camera trap images. We have worked with the Timelapse developer to integrate the output of our API into Timelapse, so a user can:\n",
    "\n",
    "- Select or sort images based on whether they contain animal or people or vehicles.\n",
    "- View bounding boxes during additional manual annotation steps (which may speed up review...)\n",
    "\n",
    "See our [Integration with Timelapse](https://github.com/microsoft/CameraTraps/blob/master/api/batch_processing/integration/timelapse.md) page for more information.\n",
    "\n",
    "![Screenshot showing the Timelapse application with MegaDetector output, shown as a bounding box around the detected animal](../api/batch_processing/integration/images/tl_boxes.jpg)\n",
    "\n",
    "\n",
    "**Subsetting the output JSON into more manageable JSONs to use with Timelapse**\n",
    "\n",
    "People using Timelapse often split up the workload and have multiple Timelapse databases per study. We wrote a simple Windows desktop app ([Camera Trap Batch API Output Manager App](https://github.com/microsoft/CameraTraps/blob/master/api/batch_processing/postprocessing/CameraTrapJsonManagerApp.md)) for splitting up the output JSON.\n",
    "\n",
    "\n",
    "#### 2. Separating images into folders that contain animals/people/vehicles/nothing\n",
    "\n",
    "Some of our collaborators do their downstream labeling without Timelapse, by moving the images to separate folders containing animals/people/vehicles/nothing according to MegaDetector output. You can use the script [separate_detections_into_folders.py](https://github.com/microsoft/CameraTraps/blob/master/api/batch_processing/postprocessing/separate_detections_into_folders.py) to do that if you have some familiarity with running Python scripts. This script has only one external package (\"tqdm\") to install - you can also follow our [installation instructions](https://github.com/microsoft/CameraTraps#initial-setup) to run the script inside a conda virtual environment. \n",
    "\n",
    "\n",
    "### Help me process millions and millions of images\n",
    "\n",
    "We offer a [service](https://github.com/microsoft/CameraTraps/tree/master/api/batch_processing#detector-batch-processing-api-user-guide) for processing a large quantity of camera trap images using the MegaDetector. We typically call this service on behalf of ecologist users once their images are uploaded. Please see our [collaborations](https://github.com/microsoft/CameraTraps/blob/master/collaborations.md) page for more information and get in touch with us."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "collapsed_sections": [],
   "name": "Megadetector_Colab_test.ipynb",
   "private_outputs": true,
   "provenance": [],
   "toc_visible": true
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}
